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Overview: Automated Vehicle Safety i", 


= Sorting out truth, myths, and “it's complicated” 
e Companies say they are safer than human drivers 
e But public trust has been eroding 


= Truth/Myth topic areas, including: 
e Are automated steering features safer? 
e Are robotaxis safer than humans yet? 
— Is that even the right question to be asking? rL Pes 
e Important misconceptions - om Heenan a 
e Other issues that still need attention 
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Why Is AV Safety Complicated? Medio 
= Public expectations ee 
e Expect super-human machine performance 
e Trust too easily given, backlash when broken 
= Technical challenges 
e Machine Learning safety is work in progress 


= Industry culture clash Bai 
e Machine Learning: 99% is a great result vs. safety is 99.9999...% 
e Silicon Valley: move fast + break things 
e Automotive: blame driver for not mitigating equipment failures 
e Regulators: test-centric; struggling with software safet 
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Robotaxis: “Safety Is Our #1 Priority” tier. 


sd cruise PF motional 
} Safety Drives Us 


Motional is developing safe 
autonomous vehicles. 


Because 
Safety is 
Urgent™ 


b] 


https://motional.com/safety-philosophy 


Our Mission Is Urgent 


https://getcruise.com/safety/ 


Autonomous Driving 


Technology Can Save ZOO XK 


Lives and Improve 


Mobility A new. bar for Safety 


https://waymo.com/safety/ Safety isn't just part of what we do. It's why we're here 
https://zoox.com/safety/ 4 
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Automated vehicle Incidents Hee Ne 
m Uber ATG fatality, Tempe AZ/US: March 2018 
e Uber ATG closed: January 2021 
= Local Motors shuttle driver injury Backup Driver Of Autonomous Uber 
e Company closed: Jan. 2022 SUV Charged With Negligent 
= Pony.Al crash, CA/US: Oct. 2021 ere -—- 
e Uncrewed test permit revoked 
= Easymile shuttle phantom braking injuries: (2019, 2020) 
= Cruise & Waymo issues in San Francisco 
e Stalling in traffic, emergency responder issues; fire truck crash 
= Cruise pedestrian dragging injury: Oct. 2023 
e Testing permits revoked; operational shutdown © 2024 Philip Koopman 5 
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Public Trust Is Eroding Cinpeaty 


Driver Attitudes Toward Self-Driving Vehicles 


2024 Survey Responses Driver Attitudes Over Time 


® Afraid 66% 2021 2022 2023 2024 
@ Unsure 25% 
@ Trust 9% ® Afraid @ Unsure @® Trust 


[AAA: https://bit.ly/48YPgZe] © 2024 Philip Koopman 6 
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Types of Vehicle Automation ee 


= Driver Assistance Pelican 


e The person drives; the car helps 


Ry 


= Supervised Automation Pipes aiaae 
e The car mostly drives; the person helps © 


e Lane Centering technology im! 


= Autonomous OPERATION. 
e The car does all the driving ani. VEHICLE 


TESTING 


= Testing es 
e Test driver compensates for automation defects 
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You Can Ride in an 
Autonomous Vehicle 
Today 
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Robotaxi Deployments vee 
= Waymo: 
e Phoenix, San Francisco, Austin, Los Angeles 
= Motional: 
e Las Vegas 
= Cruise: 


e Paused (previously multiple cities) 


= This will likely change over time 
e Other companies; other cities 


[Waymo] 
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Other pilots/deployments/testing 
e Local parcel delivery 
e Low speed shuttles 
e Full size buses 
e Middle-mile trucks 
Driver-out operations over time 
e Varies by company, 
operational concept 
Chinese robotaxis  thtps:pittystsakwe} 


e Policy seems to be continuous 
remote safety supervision, for now 


[Nuro] 
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Remote Operators Unie 


= Remote operator roles 
e Full remote driving 
e Remote safety operator 
e Remote intervention when requested 


= Remote operator and safety LAS PESTS 


a narrow street to let a buss pass. 


e Infrequent remote interaction perhaps OK cruise confirms robotaxis rely on 
-s human assistance every four to five 
— Depends on the specifics enilee 
e Can remote operator cause safety issues? 


e Can lack of remote operator request cause safety issues? 
= Many open questions here... 


hitps://bit.ly/4apOeqc 
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Autonomous 
Pilot Deployments Are 


Already On Public Roads; 


Testing Continues 


Car 
SUPERVI SED Me ‘lor ‘eo 
AUTOMATION University 


@ MYTH 


Personally Owned 
Vehicles Can Drive 
Themselves Safely 
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Personal Vehicles Require Supervision {,,, 


= Personal vehicle driving automation: 
2 is Culver City CA, 2018 [NTSB HAB-19/07] 
e Adaptive cruise control 
e Automated lane centering 
= Driver plays a role in safety 


e Limits to automation capabilities 


= So-called “Level 2/2+" systems — = 
e Hands-on: Tesla, Audi, Kia, Mercedes Benz, Volvo, Nissan, Infiniti 


e Hands-free: GM, Ford, BMW  httos:it.y/acisxa] ALTORATION 
a ” © 
= So-called “Level 3° systems (@ 


e Mercedes Benz (but driver must still monitor traffic conditions) 
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Automation complacency: Hdenewocimendoe: 400bent 
e Drivers over-trust automation 

e Attention wanders 

e Temptation to stop monitoring 


Tractor-trailer combination 
vehicle still in motion and 
completely blocking all US 441 
southbound lanes. 


Bad things can happen very quickly 
e Delray Beach fatality, 2019 

e Engagement 9.9 secs before crash 

e No human steering for 7.7 seconds 


Delray Beach, FL, 2019 
NTSB HAB-20/01 


Driver Monitoring technology might help... 
... but is still a work in progress 
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IIHS: Only 1 of 14 Systems “Acceptable” (ii. 


Driver involvement 


Driver Attention Emergency’ Lane ACC Cooperative Safety 
onitoring reminders procedures change resume steering features 


Lexus Teammate 
with Advanced 
Drive 


2022-24 Lexus LS 
SUPERVISED 
General Motors AUTOMATION 


Super Cruise M Po G G ie A iz al 


2023-24 GMC Sierra 


Nissan ProPILOT 
Assist with Navi- 


ink HHA Ff HEH HA 


2023-24 Nissan bea 
= 11 more 


[IIHS 2024; Other 11 rated “Poor” https://www.iihs.org/ratings/partial-automation-safeguards] © 2024 Philip Koopman 16 


Automated Steering 
Requires Continuous 


Human Driver Attention — 
Not Really “Self-Driving” 
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<2) Misleading 


People Are Inherently 
Terrible Drivers 


Its Complicated 
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The Myth of 94% Human Error eee 


m “94% of serious crashes are due to human error” 


— Consumer Technology Association 
Testimony to US Congress, July 2023 OR NHTS A 
[https://bit.ly/3TNMdi1]} 


Benefits of Automation 


SAFETY 


= Humans failed to prevent 4 HUMAN CAUSE’ | ivecses seis sssionsiss ees open Assn 


vehicles’ potential to save lives and reduce injuries is rooted in one 


e What the NHTSA source study actually Says: __ «vec erentersseusesiessauetonunn 


Bifor Automated vehicles have the potential to remove human error 


“The critical reason was assigned to drivers in an from the crash equation, which will help protect drivers and 


passengers, as well as bicyclists and pedestrians. When you consider 


estimated 2,046,000 crashes that comprise 94 percent more than 35,092 people died in motor vehicle-related crashes in the 


U.S. in 2015, you begin to grasp the lifesaving benefits of driver 


of the NMVCCS crashes at the national level. [DOT HS 812 115] ooo ecinciosce 


However, in none of these cases was the assignment gioor ae eO ces 
intended to blame the driver for causing the crash.” 
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Jan. 2022: 


https://bit.ly/4930UjX 


‘It Ain’t 94 Percent’: NTSB Chair Jennifer 
Homendy Discusses the Role of Human Error in 
Car Crashes 


6:01 PM EST on January 31, 2022 


20 


Carnegie 


Industry: Replace Terrible Human Drivers !*",,, 


. Kyle Vogt @ t Follow oo 
~~ @kvogt 


nM Ss CG i eS We ran this full-page ad in @nytimes and several local papers today. 
Human drivers aren't good enough. America can do better, and it is time 


we fully embrace AVs. 


terrible drivers If :--« 


42,795 Americans were killed 
in car crashes last year 


A? OS America ns were ki lled aes = 
in car crashes last year = 


You might be a good driver, but many of us arent. 
People cause millions of accidents every year in the US. 
Cruise driverless cars are designed to save lives. 
Our cars were involved in 92% fewer collisions as 


the primary contributor.* They also never drive : 
distracted, drowsy or drunk. De Et oer ok 
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Human Drivers Can Improve ee 
= Fatality/injury rate reduced: 
e Fatality/VMT: 3 
60% v —=Fatality/100M VMT 
Ini NMT Ss 2.5 —=|njury/10B VMT 
only Una = 3 —Fatality/1M population 
47% S 
e Fatality/Person ‘© 15 
67% S 1 
= 
8 
ni 


é 0.5 Source: NHTSA Traffic 
Multiple factors ; 


at work to 1985 1990 1995 2000 2005 2010 2015 2020 
improve safety YEAR 
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Might We Do Better? ee 


= Alcohol-related road fatalities: 


e US: 1985: 41% of fatalities wp ee”, 
2019: 28% of fatalities — tnutsatratticracts| 7 Ne 2 — 
e UK: 1985: 18% of fatalities https://bit.ly/4cpreS2 ; 


2019: 13% of fatalities https://bit.ly/3Tspve2 


= US fatality rates: 1985 2.50 /100M VMT tarsal 

2019 1.11 /100M vMT (1.37 in 2021) 
= UK fatality rates: 1985 2.67 /100M VMT fattuk.gov 

2019 0.51 /100M VMT (0.52 in 2021) 
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US 


Many Countries Do Better Than the US 
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Better Road Safety 


Does Not Require Using 
Computer Drivers 


TRUE 


Computer-Controlled 
Active Safety Features 
Can Improve Safety 
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Active Safety Can Really Work! ea 


= Example Warning features: 


e Back-up camera & warning Automatic emergency braking 


e Tire pressure monitoring ” 

e Rear cross-traffic alert . 

¥ 

= Example Active Safety: ¥ 


e Electronic Stability Control (ESC) 


50% 
56% 
14% 
24% 
41% 


Front-to-rear crashes 

Front-to-rear crashes with injuries 

Claim rates for damage to other vehicles 

Claim rates for injuries to people in other vehicles 
Large truck front-to-rear crashes 


e Automatic/Advanced Emergency Braking (AEB) =f 


e Lane Keeping Assistance (LKA) 


— Momentary nudge at lane boundary 
e Does NOT INCLUDE sustained steering (Lane Centering) 
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Example Car Safety Features en ee 


= http://MyCarDoesWhat.org F 
e List, icons & descriptions ‘c 


Back-up Camera Back-up Warning Rear Cross Traffic 
Alert 
Anti-Lock Braking Automatic Emergency Adaptive Headlights Bicycle Detection Brake Assist 
System Braking Lane Departure Drowsiness Alert 
: Warning 
Tire Pressure Curve Speed Warning 


; , ; 2S 
Forward Collision Left Turn Crash Obstacle Detection Pedestrian Detection Traction Control MOMEOENS aye 


Warning Avoidance © 2024 Philip Koopman 28 


Computer-Controlled 


Features 
Can Improve Safety 
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SU SED Hello 
AUTOMATION University 


® MYTH 


Automated Steering 
Improves 
Driving Safety 


7 a Carnegie 
Automated Steering Vs. Active Safety = t,, 
= Active safety: 


e Lane Departure Warning (LDW) 
e Lane Keeping Assist (LKA) 
— Momentary nudge at lane boundary 


[MyCarDoesWhat.org] 


Lane Departure Warning 


’ SUPERVISED 
= Automated steering: "aa Lane Keeping Assist 
e Lane Centering/Autosteer @ 
— Sustained steering control 


e It's not really “assist” — it is actually steering the vehicle 
- Driver is no longer continuously controlling vehicle 
— For decades we've known this causes “driver drop-out” attention loss 
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Crashes per 100 million miles 


Active Safety Makes The Difference 


Autosteer 
Adjusted 


Safety Only 


\Z , = Claimed safety 
Der a ies me benefits diminish 
Unadjusted “-." ™. aes adjusted for: 
e Active safety 
feature benefits 
a3 04 Ql Q2 a3 a4 ai Q2 a3 a4 al e Driver age 
— oe Auton, Revd ane ane aiusted 


2018 2018 2019 2019 2019 2019 2020 2020 2020 2020 2021 
e Freeway vs. other 
Active Safety Only, Road and Age Adjusted 


=— © — Autopilot, Road Adjusted roads 


— — — Active Safety Only, Road Adjusted 
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Active = Noah Goodall, 2021 
e Analyzed the data 


+++¢@+++ Autopilot, Unadjusted https://doi.org/10.31224/osf.io/m8j6g © 2024 Philip Koopman 32 


gie 


e Carne. 
Automated Steering Not A Safety Feature =", 
= 2024: Insurance Institute for Highway Safety (IIHS) 


Safety features 


There is little evidence that partial automation has any safety benefits, so it's 
essential that these systems can only be used when proven safety features are 


engaged. These include seat belts, AEB and lane departure prevention. 


IIHS: March 2024 
https://bit.ly/3Vsi35k 
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Automated Steering 
ISA 


Convenience Feature, 
Not A Safety Feature | 


Car e 
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People Are Terrible At 
Supervising Automation 


a “ Carnegie 
Automation Bias & Complacency alahite 
= Automation Bias 


e People tend to over-trust automated 
decision making 


= Automation Complacency 
e Inattention to potential malfunctions 


= Skill Degradation 
e Relying on automation degrades skills 


See: https://en.wikipedia.org/wiki/Automation_bias 
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NTSB Recommendations Melly 


= NTSB H-17-41: 
e Incorporate system safeguards that 
limit the use of automated vehicle 
control systems to those conditions for 


which they were designed. 

= NTSB H-17-42 

e Develop applications to more effectively 
sense the driver's level of engagement rahe rte 
and alert the driver when engagement is "seston Snares 
Williston FL, May 2 2016 

lacking while automated vehicle control CSTE ApS ae 
systems are in use. 

Also: H-17-37, H-17-38, H-17-39, H-17-40, H-17-43, H-20-2, H-20-3, H-20-4 
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Shape of curves will vary by system & operational concept 


Risk of Degraded Safety 


WITH ACTIVE SAFETY 19 
—~ OVERALL VALLEY OF a 
£ DRIVING DEGRADED = 
© SAFETY SUPERVISION Bota 
o . : aa =" *" DRIVERS 
Za 
Oo 
TE HUMAN ~s, AUTOMATED 
oe DRIVER . DRIVING 
D ATTENTION CAPABILITY 
® = 
= a = 
= ee tay gy = ae 


Automation Malfunction Interval (/og scale) 


HUMAN DRIVERS GiGi. 
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Driver Monitoring To The Rescue? he 


= Driver Monitoring Technology 

e Steering wheel touch sensor 

e Face & gaze camera 

e Hand position sensing 

e ... | 
= Some challenges: [Euro NCAP, 

e Sensing challenges: darkness, sunglasses, gloves 

e Intentional misuse/abuse: covered camera, wheel weight 

e Determining mental state from a person's external features 

e What if monitoring shows drivers are unable to remain attentive? 

— The real challenge is driver attention management 
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Driver 
Attention Management 


Is An 
Open Challenge 


Ordinary Drivers 
Are Qualified To 
Test Driving Automation 


Car 
Me The “i 
University 
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: Public Road Beta Testing Diora 


= Beta Testing: Operation in intended environment 
e Expectation that software can/will have defects 


A Warning [Full Self-Driving (Beta) Tesla Owner Manual] 
Model S may quickly and suddenly make unexpected maneuvers or 
mistakes that require immediate driver intervention. 


a Full Self-Driving (Beta)  '@Sla 2023.44.30.7 Release Notes 
— Last updated 23-Mar-2024 
s You can enable Full Self-Driving (Beta) by tapping 'Controls' > 'Autopilot' > 'Full Self-Driving (Beta)’ 
A5 and following the instructions. 
s Full Self-Driving is in early limited access Beta and must be used with additional caution. It may do 


the WigelateW telinte =i tel=-AWe)e a tlet=s SO you must always keep your hands on the wheel and pay extra 


attention to the road. Do not become complacent. When Full Self-Driving is enabled your vehicle will 

make lane changes off highway, select forks to follow your navigation route, navigate around other 

vehicles and objects, and make left and right turns. Use Full Self-Driving in limited Beta only if you will 

pay constant attention to the road, and be prepared to act immediately, especially around blind 
https://bit.ly/3vzWllc corners, crossing intersections, and in narrow driving situations. 42 
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Road Testing Can Cause RealHarm te, 


= Safety testing: 
e Does intended things correctly 
e Does not have unsafe surprises 
e Jesters face risk of dangerous misbehaviors | 


= Accepted industry practices 
e Simulations & test track before road test 
e Testers must have special training 
e Testing per test plan; avoid known defects / Uf 


= Ordinary retail customers should never 


“ ” SF ee Bridge Beta mutt “injury 
perform the role of tester Testing Crash, Nov. 24, 2022 
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Customers Cosplaying 
“Beta Tester” 


Expose Everyone To 
Undue Risk 


SUPERVI SED VEHICLE oath the a e 
AUTOMATION § TESTING Unig 


Blaming Drivers 
Deflects Accountability 
Away From Companies 


Carnegie 


The Moral Crumple Zone tales 


University 


= Moral Crumple Zone Strategy: 


e Human operator is a system 
component to bear the brunt 


of moral & legal responsibility 
Moral Crumple Zones: Cautionary 


Design a known unsafe system Tales in Human-Robot Interaction 
e e e t 
Deploy with a human operator ee pany 


Engaging Science, Technology, and Society (pre-print) 


System fails due to safety defect — »::..: 


Posted: 3 Apr 2016 


Blame the human operator Last revised: 15 Mar 2019 
Scrutiny deflected from defect; = Mvs""neCare Sit 


Google Inc.; University of Oxford - Oxford Internet Institute 


safety defect is not corrected _ vse written: marcia, 018 


PS ee sy 


https://bit.ly/3x8bxG 
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Autonomous Vehicle Tester Story re 


= March 2018 Uber ATG Fatality 

e Pedestrian killed during testing in Phoenix AZ 
= Complicated situation 

e Pressure to test aggressively 

e Controversy over driver behavior 
= Operator faced criminal trial 

e Plea deal to undesignated 

felony (probation) sae 

m Uber ATG faced no charges ‘I’m the Operator’: The Aftermath 


e Embarked ona safety path of a Self-Driving Tragedy “T@r™ 


https://bit.ly/3VrqnlZ 
https://bit.ly/43IcfXH © 2024 Philip Koopman 47 
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Dec. 2019: Drove 74 mph through red light 
e Off-duty limousine driver using Autopilot 


e Ran red light after end of freeway = "°s'* Mtps://bit-y/svndavT 
Note 
Autosteer is a BETA feature. 


e Killed two people in another vehicle 
Tesla faced no charges 

; As acriminal case against a Tesla driver wraps up, legal 
e Does not enforce highway-only Petrie questions on Autopilot endure 
Driver faced criminal trial : 


e Plead no contest to 
vehicular manslaughter with 
gross negligence (probation) 


No apparent industry change 


December 2023 © 2024 Philip Koopman 48 


Blaming Drivers 
Protects The Company, 


Not Necessarily 
Other Road Users 
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2) MYTH 


Lots Of Sensors 
Means No 
Avoidable Crashes 


Carnegie 


porcenten Builds the World Model = i", 


ball into street Motion Control 


10 meters ahead” 


\h) 
eS eo" 
ac 
THE REAL 
WORLD 
|z 
O COMPUTER’S 
3 WORLD MODEL: Path Planning 
Lu “Child chasing & 
O 
a 
Mm 
Oo. 


Perception & prediction 
present a uniquely difficult 
assurance challenge 
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Sensors Alone Do Not Ensure Safety {!«",,, 


“We're safe because we have LOTS of sensors!” 
= Sensor fusion 
e What if sensors disagree? 
= Perception/Prediction 
e What if system mis-classifies an object? 
e What if system mis-predicts object behavior? 
= What if there is a planning/control fault? 
e March 2023: Robotaxi hits bus 
— Detected back half of articulated bus 
— Decided to consider only front half in planning 
e April 2023: recall for software defect 
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Sensors Aren't Enough; 
Perception And 


Prediction Are 


Critical for Safety 
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2-0} Misleading 


Computers 
Wont Drive Drunk 


Carnegie 
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Aug. 2023: 

Injury crash with fire truck. 
CA DMV asked Cruise to 

cut active fleet size in half. 


__https'/(bit.ly/45fLgme6 


August 2023: Driving into Wet Concrete 
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https*//bit.ly/CruisePowerLines 


Two Cruise cars in San Francisco became wrapped in downed Muni wires and 
caution tape at Leavenworth Street and Clay Street on March 21, 2022. 


Courtesy of John-Phillip Bettencourt 
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City of San Francisco Concerns i otrarsity 


AV driving that interferes with emergency response 


Emergency Response (SFFD) Impact Incidents by Type (Jan 1 — Sept 27, 2023) 
@Cruise @Waymo 
Intrusion into operations in response zone § 


Unpredictable operations near response zone 


Contact (or near-miss) with equipment/hose 


Contact (or near-miss) with personnel 


https://bit.ly/41cwJGI 9 10 20 30 ‘oopman 57 
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Human drivers are imperfect 
e Drunk, DUI, tired } 
e Aggressively violate road rules an 
Robot drivers are imperfect 
e Software defects 

e Challenged by subtle context 
e Challenged by rare events 


e Errors in building model of fg hittbs://bit.y/artbGn 
August 2023 
the SuSE world Naat yilacthtee 
e Potential errors by Does that make this safe? 


remote human operators 
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Robot Drivers 
Will Fail - 


Sometimes Differently 
Than Human Drivers 
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Safe Enough Requires 
More Than 
“Safer Than Human Driver” 


Carnegie 


_ What People Mean By “Safe” ete 
= Human drivers are bad, so computers will be safe 
= “Safety is our #1 priority” 
= Safe driving behavior / roadmanship 
m= Tested/simulated for millions of miles 
m Risk is managed via insurance 
= Conforms to safety standards 


= Safety cases supported by evidence 
= Positive Risk Balance (better than human) 


© 2024 Philip Koopman 61 


Carnegie 


Positive Risk Balance Ringer 


= Positive Risk Balance: safer than a human driver 
= But which human driver? 
e 28% Alcohol/driving under influence fatalities 4 
e 26% speed-related, 9% distracted, 2% drowsy _ 
e 60 year old driver is ~3.5x better than 16 y.o. 
= Where/Who? 
e 3.4x fatality per VMT variation by US state 
e Victim demographic (e.g., pedestrians) 
= Which vehicle? 
e New cars have active safety — BUT average car age ~12 years 


[DOT HS 813 060 & DOT HS 813 021] [AAA] [IIHS Fatality Fact Sheets State by State] [DOT HS 813 060] © 2024 Philip Koopman 62 
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Other Safety Considerations eee 


= Avoid risk transfer to vulnerable populations 
e What if vulnerable road user risk increases? 
= Avoid negligent driving behavior 
e What if breaking traffic rules leads to crashes? 
= Fine-grain regulatory control of risks 
e Recalls due to specific risk, not net risk 
= Ethical & equity concerns 
e What if some demographics are at increased risk? 
= Potential for crash-by-crash comparison - ee 
e What if “a human driver would never have made that mistake”? 
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Need More Than 


Improved Statistical 


Average Safety 
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Insurance Cost Pressure 
Will Ensure Acceptable 
Automated Vehicle Safety 


Carnegie 
__ Insurance Leverage for Safety vel 
= 2020 US Insurance Losses 
e Total $135B 
e 40% injury/medical losses 


US 2020 Car Insurance Losses 


= 2020 Statistics 
e 2.9 Trillion vehicle miles 
e 267,585,097 Vehicles 


e 6,773,962 Collision Claims = Bodily Injury = Personal Injury 
e 810.000 Vehicle Thefts = Medical Payments = UUM Bodily Injury 
? By = Collision = Property Damage 
® 38,824 Fatalities = UUM Property = Comprehensive 
= Not all fatalities pay out big claims [Data Source NAIC https://bit.ly/3TrWHm 1 ] 
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Affordable Insurance vs. Safety ere 


m “We are safe because we bought insurance” 
e Small numbers of vehicles limits exposure 
e Insurance company maximum payout: policy limit 


= Affordable risk might exceed everyday safety 
e E.g., Life insurance for combat military personnel 


= Insurance is about pricing risk, not ensuring safety 4. 
e Customers pay for increased risk via premiums 
e Risk uncertainty perhaps more important to insurers 


Affordable Insurance # Acceptable Safety 


https: //bit. ly/4oumY8J 
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2. Carnegie 
_ Net Risk Alone Is Not Safety new 

= Redistribution of harm 
e What if more pedestrians, cyclists die? 


e What if more mishaps happen in historically 
disadvantaged areas? 


= Negative risk externalities 
e Blocking fire trucks, ambulances 

= What if known significant risks unmitigated? 
e Even if total fatalities decrease, is that OK? 

= Fatalities due to breaking traffic rules R S K 
e Humans break rules too... 


but they are held accountable via negligence 


% 
7) 
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Insurance Pressure 


Alone Will Not Ensure 
Acceptable Safety 


Carnegie 
AUTONOMOUS Mellon 
OPERATION University 


2) MYTH 


Autonomous Vehicle 
Ethics Is All About 
The Trolley Problem #\gur 


Given a no-win situation, 

should the vehicle: 

e Kill 1 person to save 5? 

e Kill socially devalued people 
— Safety only for suit-wearers? 

a This is a false dilemma! 

e How often will this happen? 


e Why was the car not equipped / 


with redundant brakes? 


e@ Why did the car not roll itself 
over using a side barrier? 


https://www.moralmachine.net/ 
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In this case, the 
self-driving car with 
sudden brake 
failure will continue 
ahead and drive 
through a 
pedestrian crossing 
ahead. This will 
result in ... 
Dead: 

e 2 homeless 

people 


Note that the 
affected 
pedestrians are 
flouting the law by 
crossing on the red 
signal. 
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Ethics: Deployment Governance ie 


= #1 ethical issue is deployment governance 
e Who decides when to deploy based on what? 


= Aggressive for-profit deployments 
e Existential financial & time pressure 


e Missing independent technical oversight : ALC 


= Ethical deployment should address: 


e Publicly disclosed safety prediction 
e Inclusion of stakeholder concerns 
e Transparency of data & processes 
e Accountability for any losses 
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Equity Concerns en We 
= Ride Hail made promises ... with disappointing results 
e Why will for-profit robots turn out differently? 
m= Labor concerns: 
e Displaced ride-hail/taxi drivers 
e Displaced truck drivers 
i) Transportation access concerns: 
e Service for disabled in absence of regulations? 
e Cheap taxis undermining safer public transit 
= Risk distribution concerns: 
e Testing risk might be imposed upon vulnerable people 
e Municipal preemption / no local control of issues epooa rnikeepmeney3 


Ethics/Equity Question: 


Who Decides 
What / When / Where 
To Deploy 
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AUTONOMOUS Me ior a 


oo MYTH 
10 Million Good Miles 
Has Proven 
Autonomous Vehicles 
Are Safe 
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2023: Results From 1M+ Miles oe 


Waymo: Feb. 2023. 
In January 2023, https:/bit ly/3NSF6xF Sept. 2023 _sihttps://bit.ly/43KNmKZ 


Waymo reached 1 million rider-only miles Waymo + Swiss Re Report 
ee 3K Bas. Emphasis Based on 3.8 million miles 


No reported injuries Only 2 collisions that met the 18 minor contact events 
criteria for inclusion in NHTSA’s CISS 


1, +, 
ais a S on Qe 
55% of all events were the result of Human drivers violated road rules 10% of all events happened 
a human driver hittinga stationary and/or behaved dangerously in at night aa yy 
Waymo vehicle every vehicle-to-vehicle event al t al U t 


uF Re 
No intersection-related events No events involving vulnerable h 
id 
ee crasnes Safer than human- 
Updated Human Ridehail Benchmark vs Cruise AVs in IM driven vehicles. 
Collision C in San Fi i 
acai catia Cruise: Sept. 2023. With 100% fewer bodily injury claims and 76% fewer 
60 https://bit.ly/47W1DVR property damage claims, Swiss Re (one of the 


world’s leading reinsurers) concluded that Waymo 


is significantly safer than human-driven vehicles. 


60 


Waymo as of March 2024: 
https://waymo.com/safety/ 


40 


20 


Collisions in 1 Million Miles 


Waymo passenger injury August 2, 2023 -- 


Collisions Collision with Collision with 


Primary Contribution Meaningful Risk of Injury the day after Swiss Re study decided to 
sn Hv SsieneeiueiSiannsatmcs end data studied: https://bit.ly/47Z9pyb 


fm Human Ridehail Benchmark (Refined Estimate) 
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_How Many Road Testing Miles? =", 


= Human driver miles per fatal crash: inursa 
e US: 1999: 98M VMT / 2021: 79M VMT 
e Includes drunk, impaired, speeding, ... 
= Statistically good as average human driver 


e 95% confidence Tens of millions of miles. 
ee oT Segre ne SUE IDO Dafety apie eelanarre 
— But at this point you likely have fatal crash(es)... the Moon and back 80 times. 


Including test driver miles. 


° H Waymo as of March 2024: 
e Rule of thumb: need 10x miles per crash Sa a Lae 


= Waymo 7.1M mile report: joec. 2023 at page 15; https://bit.ly/4cDuZvs] 
e “no statement...can be made’ regarding serious injury/fatalities 


https://reliabilityanalyticstoolkit.appspot.com/mtbf_test_calculator © 2024 Philip Koopman 77 
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Are Robotaxis Safer? ee 


University 


= Robotaxi companies predict acceptable safety 
e Based on non-severe crash rates 
e With sometimes controversial limitations Our Safety 
e Fatality & serious injury rates are predicted BOSoR ny 
= 300+ Million miles needed to confirm eben peat 


already reducing traffic injuries and fatalities in 


oune ° 0 he places where we currently operate. At Waymo, 
e Perhaps 5-10 million driverless miles now [ee ree, 


we aim to reduce traffic injuries and fatalities by 


e With continually evolving software ecb te la eiehaes 
e Reduced fatality rates are still aspirational 


[https://waymo.com/safety/] 


= Declaring safety “victory” at this point is like claiming a medal 
... after the first mile in a marathon 
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Companies Predict 
— But Cannot Yet Prove - 


Severe Injury/Fatality 


Safety 
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2) MYTH 


Road Testing Makes 
Autonomous Vehicles 
Safe 


How About A Robot Driver Test? Meio 

= Written test for Automated Driving System (ADS) 

e Does ADS know traffic laws & behaviors? 
= Road test 

e Can ADS obey traffic laws? 

e Can ADS negotiate effectively with human drivers? 

e Can ADS resolve potentially ambiguous situations? 
= Being a 16 year old human 

e How do we measure ADS judgment maturity? 

e Autonomous systems struggle with novelty, unknowns 
=» Need safety engineering, not just a driver test 
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Brute Force Road Testing ie 
a If 100M mil es /fatality... _ % WolframAlpha vez. 
e Test 3x-—10x longer than mishap rate Src 
=> Need 1 Billion miles of testing > 


= That's ~50 times on 


every road in the world eS 
e With fewer than 10 fatalities = aoe > 


e Start over for each software update(?) 


© 360000 t 720000 © 1.4 million t& 1.8 million 
0 720000 1.1 million @ 1.8 million t 2.1 million 
14 t 360000 © 1.1 million © 1.4 million M& » 2,1 million 


= Brute force testing impracticable | 
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Have you covered the beseltic Pu EIS 


| 
ea 


Ring road 
Cirencester 
A4289 x 


= 
J Marlborough 

Town Burford 

centre Pr Oxford 


ANE: 


http://bit.ly/2tvCCPK 
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Good prediction based on the world model 

e Classification accuracy affects prediction 

e Multiple possibilities for any object in any situation 
Safety limited by heavy tail scenarios (rare + important) 


e Probabilities of what 
happens next are 
context dependent 


Rare cases/unusual 
context can dominate | =a  - ia 
safety cas _— a : a oily/3edsB07 


pit ly/3SSuaEQe * - © 2024 Philip Koopman 84 


PROBABILITY OF SURPRISE 


Carnegie 


Heavy Tail Distribution Of Surprises ae 


Common Things Edge Cases 


: Seen In Testing . Not Seen In Testing 


Random Independent Arrival Rate (exponential) 

r Law Arrival Rate (80/20 rule : 

= Tail Condi Many Different, 
Infrequent Scenarios 
Total Area is the same! 


TOTAL TESTING TIME ——————_> 


© 2024 Philip Koopman 85 


University 


= Where will you be after 1 Billion miles of testing? 


= Assume 1 Million miles between unsafe “surprises” 
e Example #1: 
100 “surprises” @ 100M miles / surprise 
— All surprises seen about 10 times during testing = , 
— With luck, all bugs are fixed 


e Example #2: Heavy Tail 
100,000 “surprises” @ 100B miles / surprise - { 
- Only 1% of surprises seen during 1B mile testing ay Missle 
- Bug fixes give no real improvement (1.01M miles / surprise) 
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Safety Engineering In A Nutshell Melo 
= Safety Engineering Process 
Identify hazards 
Determine risk from hazards 
Mitigate risk from hazards 
Repeat until acceptable remaining risk 


= Open challenges 
e How heavy tail is the distribution of event types? 
e Applying safety engineering to machine learning 
e How much/what type of remaining risk is acceptable? 
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Safety Depends On 


Engineering To 
Mitigate Rare, High- 
Consequence Events 


Heavy-Tail Distribution 


Of Surprises 
Is A Challenge To 
Scalable Deployment 
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2) MYTH 


Safety Standards 
Don't Exist and/or 
Would Stifle Innovation 
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Standards Set Expectation of Safety i, 
SYSTEM ANSI/UL Seley age 
SAFETY 4600 eS 
Driving HIGHLY 
AUTOMATED 
DYNAMIC = ISO. —sSaFAD/ISO._ Environment & J VEHICLE 
DRIVING 21448 TR4804_ _—_ Edge Cases ae 
FUNCTION CASE 
ANSI/UL 
FUNCTIONAL __|SO Equipment 4600 
SAFETY 26262 Faults 
ROAD 
CYBER- SAE SAE Computer TESTING 
SECURITY J3061 21434 Security SAFETY 
SAE 
VEHICLE eaele J3018 
SAFETY Vehicle 
Functions 
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Salely standars & ingevallen, 5 uta 


University 


AV Industry: standards/regulation “Stifle Innovation” 
= Do safety standards mandate particular technology? 
e NO - they require engineering rigor to show safety . 
= Do safety standards limit ability to test prototypes? _ 
e NO - primarily apply to public road deployment | 
= How do safety standards limit ability to road test? | 
e Use of trained safety drivers and test plans 
e Big Red Button to disable computer control must actually work 
= The burden for testing innovative approaches is minimal 
e Removing the safety driver is deployment, not safety testing 
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Case Study: Loss of Titan Submersible 


University 


OceanGate was also concerned that the classing process could slow down 
development and act as a drag on innovation. “Bringing an outside entity up 
to speed on every innovation before it is put into real-world testing is 


anathema to rapid innovation,” it said. 


In an interview with the Smithsonian magazine in 2019, Rush complained 
that the commercial sub industry had not “innovated or grown - because 


they have all these regulations”. The Guardian https://bit.ly/3PuM291 


= Catastrophic 2023 implosion 
e Unorthodox construction techniques 
e Did not submit to external safety review 
e Developer attitude: 


— Real world testing is what matters 
—- Regulation kills innovation 


Missing Titanic Submersible 
‘Catastrophic Implosion Likely Killed 
5 Aboard Submersible 


Pieces of the missing Titan vessel were found on the ocean floor, 
about 1,600 feet from the bow of the Titanic, the Coast Guard said. 
OceanGate Expeditions, the vessel’s operator, said, “Our hearts are 
with these five souls.” 


Published June 22, 2023 Updated June 26, 2023 


faa] Share full article aR ml 


Coast Guard Says Debrisof Submersible. Has Been Found 
‘\ 


~ ~—_ 
i 


1:18 


The U.S. Coast Guard said parts of the Titan submersible found on the ocean floor 
indicate a “catastrophic implosion” of the vessel. OceanGate Expeditions, via Associated Press 
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Safety Standards 
Deter UNSAFE 


Innovation 


Government Regulation 
Will Ensure Safe 
Vehicle Automation 


Me hs a . 
University 
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Robotaxi Regulatory System In Action — tk", 


Los Anacles Times = October 2, 2023 crash 
General Motors recalls all Cruise e Human-driven 
robotaxis after one dragged a pedestrian vehicle hits pedestrian 


e Cruise runs over person 


e Cruise robotaxi drags 
person after initial stop 


e Regulator interactions 


e Oct. 24, CA DMV 
suspends Cruise permits 


e Nov. 7, NHTSA Recall for 
post-collision response 


General Mo er rte ti pi naan of its Cruise robotaxi vehicles —— ce pe esi ieee a pedestrian in San Francisco 
fest nia ing to doc osted by s ely eaulator s Wedne as oe ose iated Pre a 
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BY TOM KRISHER | ASSOCIATED PRESS NOV. 8, 2023 8:32 AM PT 


Carnegie 
; US Regulatory Posture geste 
= Federal / equipment safety: reactive (recalls) 
e NHTSA 2020 proposal to use industry standards spilles 
e Started collecting “SGO” crash datain 2021 fay 


= State / driver safety: administrative only 
e Texas, Arizona, etc. “open for business” 
e California: permits, licensing, reporting 
— But — impossible to ticket a robotaxi 


https://goo.gl/dBdSDM 


= Municipal / adapt to locality: frustration 
e State preemption of localities 
e Pushback starting after San Francisco experiences 
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Regulators Struggle with Novel Technology *",,, 


= Regulatory recalls 
e “Undue Risk” in the small - specific issues 


e Informed by test-centric standards BL 
= Recalls historically specific, not net risk NHTSA 
e Rolling through stop signs wired niguwhy eA cE 


@ Phantom braking SAFETY ADMINISTRATION 


e Malfunctioning display console 


= Regulators struggling to predict safety outcomes in advance 
e Software safety & net risk are historically beyond regulatory scope 


Part 573 Safety Recall Report 
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Feb 2022: Feb 2024: 
Tesla recall: ‘Full Self-Driving’ software CR's Extensive Testing Shows That 
runs stop signs Tesla's Autopilot Recall Fix Does 


Not Address Safety Problems 


The changes to warning messages and controls 
don't go far enough to prevent misuse and 
distraction, CR's car safety experts say 


FILE - A 2021 Model 3 sedan sits in a near-empty lot at a Tesla dealership in Littleton, Colo. June 
27, 2021. Tesla is recalling nearly 54,000 vehicles because their “Full Self-Driving” software lets 
them roll through stop signs without coming to a complete halt. Documents posted Tuesday, Feb. SEE SSS 
1, 2022, by U.S. safety regulators say that Tesla will disable the feature with an over-the-internet CR's Tesla Model Y 
software update. (AP Photo/David Zalubowski, File) https://bit ly/43xeX27 Photo: John Powers/Consumer Reports https://bit.ly/3voV9B8 99 
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Federal Recall-Based 


Strategy Struggling 
To Deal With 
System-Level Safety 


Product Liability 
Will Ensure Safe 
Vehicle Automation 
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Product Liability Is Not Enough vane 


= Manufacturers are pushing for only product liability 
e Manufacturing defect, design defect, etc. 
‘ Mercedes To Accept Liability When 
e Must prove product presents undue risk autonomous Drive Pilot Is Engaged 


Drive Pilot is a Level 3 system, and Mercedes will be the first 
automaker to accept legal responsibility when sucha 


= Difficult and expensive to prove nese 
e Source code analysis expensive + painful | 
e Class action requires commonality 
— With weekly neural network updates? 
e Poor machine learning explainability? 


= Does this make sense if the 
car ran a red light and crashed? 
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Product Liability ls The 
Wrong Tool For Most 


Automated Vehicle 
Crashes 


Current Tort Liability 
Rules Will Ensure Safe 
Vehicle Automation 


ee Carnegie 
Tort Law for Non-Specialists tere 
= Civil Tort Law 
e Compensate a claimant who has suffered loss ... 
proximately caused by ... 
the negligence of another party. 


= Key idea: Duty of Care 
e A human driver has Duty of Care to other road users » 
— Breach of this duty of care > negligence 
e Must act as a “reasonable person” would act Sy 
— A theoretical competent, unimpaired person, according to a jury 
— Per incident > statistical safety does not avoid negligence 
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Duty of Care for Accountability =", 


= Legal fiction of a “computer driver” 
e Sustained automated steering of vehicle 
e Manufacturer is responsible 
= Transfer of duty of care is key 
e Computer driver has it while steering 
e Can transfer duty of care back to human 
— With sufficient notice ee 
= Computer driver held to same standard as human driver 
e Would a human driver have been negligent? 
— Loss resulting from traffic law violation is negligence per se 
e Statistical safety doesn’t avoid negligence (no “free hits”) 
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Implications of Defining a Computer Driver ‘i, 
= Most crashes can be handled by tort law 
e Computer Driver that runs a red light ... 
.. held to same rules as if a Human Driver 
— Do we really need source code analysis for this? 
e Avoids overwhelming courts with product liability 
— Straightforward fix without rewriting existing law 
e Analogous to “electronic signatures” = signatures 
= Financial pressure for safe driving behavior 
e Same rules for Computer & Human Driver behavior 
e Manufacturer bears costs from any unsafe driving 
e Need more for acceptable safety at scale! But this /s a start. 
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Alternative to SAE Levels for Regulation = (i, 


DRIVER H ° 4 
ieciec ce = Conventional: Human Driver steers 
@ e Human Driver responsible 


autonomous } & Fully Autonomous: Computer Driver steers 
OPERATION x < ° 
e Manufacturer is responsible for Computer Driver 


oo 


VEHICLE = Testing: Development, Beta, Pre-production 


TESTING 
e Manufacturer is responsible for safe test plan, 
qualification and performance of test drivers 
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The Awkward Middle ae 
m Unify SAE Levels 2/3 into single regulatory bin 
e Computer steers + other control; human supervises 
= Activated computer driver accepts duty of care 


SUPERVISED 
e Human role determined by operational concept AUTOMATION 
= Computer driver can relinquish duty of care: © 
1. Due to driver monitor violation ay 


2. Due to exiting Operational Design Domain 
e But only after 10 second minimum safe harbor for human driver 
— Best effort fault mitigation after 10 second timer 
— Longer safe harbor if jury says this is reasonable for situation 
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Providing A Safety Guardrail ee 
= Automated steering is the key safety attribute 


= Net risk metrics are insufficient 
e Safer than human is a long term goal 
e Will take years for equipment regulations 
e What about risk redistribution & inequities? 
e Solutions needed, but will take time 


= Computer Driver concept 
e Compatible with what many companies are selling 
e Imposes same requirements we already use for human drivers 
e Holds companies accountable for cost of mishaps 
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Tort Law Could Help 
Support Safety — Via 


Computer Driver 


Concept 
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Essential Vehicle Automation Safety | ee 


1. Safe as a human driver on average 
e Perhaps 100M miles/fatal crash (better for good drivers) 

2. Avoiding risk transfer onto vulnerable populations 
e Pedestrian harm should not increase even if net Heyl is neotcey 

3. Avoid negligent computer driving ‘(||| Agate (|| 
e Running red lights and stop signs is not OK Billi eae 

4. Conform to industry safety standards; 
e Uncrewed operation = deployment 

5. Address other ethical & equity concerns 
e Limited local authority; manufacturer accountability for harm 
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Resources Diet 


m Video lecture series on autonomous vehicle safety: 
e Keynote talks: https://users.ece.cmu.edu/~koopman/lectures/index.html#talks 
e Mini-course: https://users.ece.cmu.edu/~koopman/lectures/index.html#av 
m “Safe Enough” book & talk video: 
e https://safeautonomy.blogspot.com/2022/09/book-how-safe-is-safe-enough- 
measuring.html 
m UL 4600 AV safety standard book & talk video: 
e https://safeautonomy.blogspot.com/2022/11/blog-post.html 
m Liability-based proposal for state AV regulation & podcast 


e https://safeautonomy.blogspot.com/2023/05/a-liability-approach-for- 
automated.html 


m US Congressional House E&C testimony: 
e https://safeautonomy.blogspot.com/2023/07/av-safety-claims-and-more-on-my.html 
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